Finite Precision Error Analysis of Neural Network Hardware Implementations

29 computation. On the other hand, for network learning, at least 14-16 bits of precision must be used for the weights to avoid having the training process divert too much from the trajectory of the high precision computation. References [1] D. Hammerstrom. A VLSI architecture for high-performance, low cost, on-chip learning. Figure 10: The average squared dierences between the desired and actual outputs of the XOR problem after the network converges. The paper is devoted to the derivation of the nite precision error analysis techniques for neural network implementations, especially analysis of the back-propagation learning of MLP's. This analysis technique is proposed to be more versatile and to prepare the ground for a wider variety of neural network algorithms: recurrent neural networks, competitive learning networks, and etc. All these networks share similar computational mechanisms as those used in back-propagation learning. For the forward retrieving operations, it is shown that 8-bit weights are sucient to maintain the same performance as using high precision 27 ture of the XOR problem, a soft convergence is good enough for the termination of training. Therefore, at the predicted point of 12-13 bits of weights, the squared dierence curve dives. Another interesting observation is worthwhile to mention: the total nite precision error in a single iteration of weight updating is mainly generated in the nal jamming operators in the computation of the output delta, hidden delta, and weight update. Therefore, even though it is required to have at least 13 to 16 bits assigned to the computation of the weight update and stored as the total weight value, the number of weight bits in the computation of forward retrieving and hidden delta steps of learning can be as low as 8 bits without excessive degradation of learning convergence and accuracy.